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Demonstration  of  a  Coupled 
Watershed-Nearshore  Model 

'.  Cheng,  Robert  M.  Hunter,  and  Hsin-Chi  Lin 


PURPOSE:  This  technical  note  documents  a  research  demonstration  of  a  newly-developed 
watershed-nearshore  computational  model,  which  couples  the  parallel  WAterSHed  systems  of 
one-dimensional  (1-D)  stream-river  networks,  two-dimensional  (2-D)  overland  regimes,  and 
three-dimensional  (3-D)  subsurface  media  (pWASH123D)  model  with  the  2-D  ADvanced 
CIRCulation  model  for  oceanic,  coastal,  and  estuarine  waters  (ADCIRC).  This  task  was  spon¬ 
sored  by  the  System-Wide  Water  Resources  Program  (SWWRP).  It  was  conducted  through  the 
simulation  of  water  flow  in  a  simplified  Biloxi  Bay  hydro-system. 

BACKGROUND:  Coupling  pWASH123D  (Cheng  et  al.  2006,  2007)  and  2-D  ADCIRC 
(ADCIRC  2009)  using  the  Earth  System  Modeling  Framework  (ESMF)  (ESMF  2009)  is  the  task 
of  the  BEI-04  project  in  the  Battlespace  Environments  Institute  (BEI)  that  is  sponsored  by  the 
DoD  High  Performance  Computing  Modernization  Program  (Department  of  Defense  2010).  The 
two  models  are  coupled  in  a  concurrent  mode  with  time  lagging,  where  the  ESMF  and  DBuilder 
(Hunter  and  Cheng  2006)  are  used  to  provide  the  software  architecture  and  parallel  data  man¬ 
agement,  including  all  the  functional  support  of  data  exchange  between  the  two  models.  The 
coupled  model  passed  the  beta  test,  where  four  critical  technical  parameters,  including  accuracy, 
portability,  integration,  and  scalability,  were  examined  according  to  the  development  and 
evaluation  plan.  This  technical  note  presents  the  coupled  model  through  a  test  example,  where  a 
simplified  watershed  and  nearshore  hydro-system  around  the  Biloxi  Bay  area  in  Mississippi  was 
considered.  This  Biloxi  Bay  model,  constructed  with  the  DoD  Groundwater  Modeling  System 
(GMS)  (USACE  2009),  used  historical  data  as  well  as  the  simulation  results  from  a  large-scale 
2-D  ADCIRC  that  modeled  the  2005  hurricane  Katrina  as  the  driving  forces  for  water 
movement. 

pWASH123D.  pWASH123D  is  the  parallel  version  of  the  WASH123D  model  that  is  a  physi¬ 
cally-based  finite  element  numerical  model  computing  water  flow  within  watershed  systems  that 
can  be  simulated  as  combinations  of  1-D  channel  networks,  2-D  overland  regimes,  and  3-D  sub¬ 
surface  media  (Yeh  et  al.  2006).  The  interactions  between  different  modeled  media  (i.e.,  between 
channel  and  overland  and  between  surface  and  subsurface)  impose  flux  continuity  and  state  vari¬ 
able  continuity  on  the  medium  interfaces.  The  pWASH123D  model  aims  to  efficiently  simulate 
the  medium-  to  large-scale  problems  on  HPC  machines.  Different  parallel  algorithms  and  parti¬ 
tioning  strategies  are  implemented  in  different  computational  components  (e.g.,  1-D  channel 
flow,  2-D  overland  flow,  and  3-D  subsurface  flow)  to  maintain  load  balance  and  reduce  commu¬ 
nication  overhead  (Cheng  et  al.  2007).  The  model  is  a  mixed  C,  Fortran,  and  C++  parallel  code 
(Cheng  et  al.  2006). 
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ADCIRC.  The  ADCIRC  model  is  a  coastal  circulation  and  storm  surge  model.  It  is  a  system  of 
finite  element  computer  programs  for  solving  time  dependent,  free  surface  circulation  and  trans¬ 
port  problems  in  two  and  three  dimensions.  The  model  implements  the  continuous  Galerkin 
finite  element  method  based  on  the  Generalized  Wave  Continuity  Equation  (GWCE).  It  is  a  par¬ 
allel  Fortran  90  code.  Typical  ADCIRC  applications  have  included:  (a)  modeling  tides  and  wind 
driven  circulation,  (b)  analysis  of  hurricane  storm  surge  and  flooding,  (c)  dredging  feasibility  and 
material  disposal  studies,  (d)  larval  transport  studies,  and  (e)  near  shore  marine  operations.  For 
the  model  coupling  in  BEI-04,  the  two-dimension  version  of  ADCIRC  is  employed. 

ESMF.  The  ESMF  is  computer  software  for  building  and  coupling  weather,  climate,  and  related 
models.  It  provides  and  defines  a  software  architecture  for  composing  complex,  coupled  model¬ 
ing  systems  and  includes  data  structures  and  utilities  for  developing  individual  models.  It  con¬ 
sists  of  a  superstructure  that  can  be  assembled  into  user  applications  and  an  infrastructure  for 
building  model  components.  It  was  originally  funded  by  the  National  Aeronautics  and  Space 
Administration  (NASA)  to  support  climate  and  weather  modeling.  The  ESMF  effort  then 
brought  together  different  areas  of  research  funding  to  extend  its  support  to  diverse  modeling 
works.  It  is  the  designated  tool  to  conduct  model  coupling  in  the  BEI.  Sponsored  by  BEI,  the 
ESMF  has  been  developing  its  unstructured  mesh  functionality.  The  coupling  of  pWASH123D 
and  2-D  ADCIRC  is  the  first  application  of  unstructured  mesh  in  ESMF.  Although  the  coupled 
model  has  used  DBuilder  for  data  exchange  through  the  coupling  interface,  the  ESMF  unstruc¬ 
tured  mesh  functionality  has  been  developing.  The  ESMF  is  still  used  for  startup,  oversight  of 
model  runtime,  and  completion  of  the  coupled  model  (Figure  1).  The  developed  ESMF  unstruc¬ 
tured  mesh  support  is  now  tested  and  incorporated  into  the  coupled  pWASH123D-ADCIRC 
model  in  the  BEI-04  project. 

DBuilder.  The  DBuilder  toolkit  is  a  parallel  data  management  library  for  scientific  applications, 
which  is  currently  used  to  facilitate  the  data  exchange  between  the  two  models  (Hunter  and 
Cheng  2005).  It  has  been  developed  to  assist  developers  in  implementing  parallel  versions  of 
their  codes  by  providing  a  simple,  consistent  application  programming  interface  (API)  that  hides 
many  of  the  programming  details  associated  with  domain  partitioning,  parallel  data  management, 
domain  coupling,  and  invoking  parallel  linear  solvers.  Because  DBuilder  supports  coupling 
independent  domains  in  a  single  model  (i.e.,  pWASH123D’s  1-D,  2-D,  and  3-D  domains), 
adopting  it  to  exchange  data  between  models  was  not  a  large  task.  It  was  used  in  the  coupled 
model  because  the  ESMF  was  lacking  support  for  unstructured  meshes  at  the  time  of  initial 
development. 

DoD  GMS.  GMS  is  one  of  the  most  sophisticated  groundwater  modeling  environments  avail¬ 
able  today.  It  provides  an  integrated  and  comprehensive  computational  environment  for  simu¬ 
lating  subsurface  flow,  contaminant  fate/transport,  and  the  efficacy  and  design  of  remediation 
systems.  It  also  provides  a  comprehensive  graphical  environment  for  numerical  modeling,  tools 
for  site  characterization,  model  conceptualization,  mesh  and  grid  generation,  geo-statistics,  and 
sophisticated  tools  for  graphical  visualization.  There  is  a  pWASH123D  graphic  user  interface 
included  in  GMS  6.5.  In  this  demonstration  task,  GMS  was  also  used  to  generate  the  ADCIRC 
mesh.  The  generated  geometry  file  was  later  converted  to  the  ADCIRC  format. 
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Figure  1 .  The  implementation  strategy  for  the  coupled  pWASH123D-ADCIRC 
model. 


METHODOLOGY:  Figure  2  shows  the  diagram  of  data  exchange  between  pWASH123D  and 
2-D  ADCIRC  in  the  coupled  model.  Through  the  coupling  interface,  the  computed  water  surface 
elevation  from  2-D  ADCIRC  is  used  as  a  Dirichlet-type  boundary  condition  for  pWASH123D 
computation,  while  the  mass  flow  computed  from  pWASH123D  is  used  as  a  flux-type  boundary 
condition  for  2-D  ADCIRC  computation. 


Figure  2.  Data  exchange  in  the  coupled  pWASH123D-ADCIRC  model. 


The  ESMF  requires  that  a  model  application  code  consists  of  three  distinct  phases:  initialize,  run, 
and  finalize.  The  application  developer  needs  to  implement  a  main  program  including  the  three 
phases,  in  which  some  ESMF  functions  are  called  (Figure  3).  We  will  call  this  file  COSM.f.  A 
user  would  then  create  an  interface  fde  that  contains  the  three  functions  (initialize,  run,  and 
finalize)  for  a  particular  model.  For  this  example,  we  will  call  them  pWASH123.c  and 
ADCIRC. f.  The  last  piece  the  user  needs  to  implement  is  a  coupler  component.  This  component 
contains  coupler  initialization  routines,  which  may  be  used  to  create  import  and  export  states 
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containing  scalars  and  vectors  for  data  exchange.  The  coupler  component,  which  is  named 
Coupler.f  here,  also  contains  the  run  routine  that  executes  the  designated  coupling  algorithms 
and  a  finalize  routine. 


Figure  3.  ESMF  coupling  and  coupler  components. 


In  the  current  implementation  of  the  coupled  pWASH123D-ADCIRC  model,  the  coupler  com¬ 
ponent  relies  on  the  functionality  of  DBuilder  for  implementation  of  the  coupler  “inif  ’  and  cou¬ 
pler  runtime  routines.  The  coupler  contains  a  key  component  provided  by  DBuilder,  which  is  an 
element  searching  routine.  When  the  models  are  run,  there  is  no  static  information  with  regard  to 
the  mapping  of  nodes  to  elements  along  the  boundary  interface  of  the  two  models.  This  interface 
boundary  may  also  be  an  overlapped  region.  At  runtime  each  model  determines  its  interface 
nodes  based  upon  its  own  boundary  condition  values.  Then  in  the  coupler  initialization  routine, 
each  model  passes  the  geometric  coordinates  of  its  interface  nodes  to  the  other  model.  The  ele¬ 
ment  searching  routine  is  then  called  on  each  model  to  build  a  list  of  elements  containing  the 
coordinates  from  the  other  model. 

One  should  keep  in  mind  that  this  is  all  done  in  parallel  over  already  partitioned  meshes  in  both 
models.  The  element  searching  algorithm  is  constructed  using  an  Alternating  Digital  Tree  (ADT) 
(Bonet  and  Peralie  2005)  with  a  complexity  of  O  (log  N),  where  N  is  the  number  of  elements. 
Once  the  element  has  been  determined,  weights  are  calculated  for  each  associated  node  of  the 
element.  These  weights  are  the  nodal  contribution  from  each  node  to  the  value  calculated  at  the 
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geometric  coordinate.  The  computed  value  is  then  shipped  back  to  the  model  processor  that  owns 
the  node. 

Data  Exchange  through  Coupling  Interface.  The  current  coupling  approach  uses  the 
coastal  (shore)  line  as  the  coupling  interface.  Figure  4  depicts  a  side  view  perpendicular  to  the 
coupling  interface.  The  current  coupling  requires  the  minimum  water  depth  option  in  ADCIRC 
to  be  activated.  In  other  words,  the  wetting-drying  scenario  is  not  handled  at  this  point.  To  ensure 
the  applicability  of  this  option,  an  elevation  drop  from  pWASH123D  to  ADCIRC  on  the  inter¬ 
face  boundary  is  enforced,  as  shown  in  Figure  4.  The  water  surface  elevation  on  the  coupling 
interface  that  ADCIRC  simulates  is  used  as  the  boundary  condition  for  computing  channel  flow 
and  overland  flow  in  the  surface  water  system.  With  the  hydrostatic  assumption  applied  to  each 
horizontal  location  on  the  coupling  interface,  this  computed  surface  elevation  is  distributed  to  the 
subsurface  nodes  beneath  the  coastal  line  for  computing  subsurface  flow.  On  the  other  hand,  the 
boundary  flow  associated  with  all  of  the  pWASH123D  nodes  on  the  coupling  interface  are  com¬ 
puted  and  then  distributed  to  the  ADCIRC  boundary  nodes  on  the  coupling  interface.  The 
pWASH123D  boundary  nodal  flow  includes  contributions  from  both  surface  and  subsurface 
flow.  In  Figure  4,  fl  represents  the  boundary  flux  due  to  surface  flow,  and  f2  the  subsurface 
flow.  In  reality,  part  of  f2  passes  through  Zone  A  and  enters/exits  the  ADCIRC  model  across  the 
bathymetry  line  as  sinks/sources.  In  this  case,  Zone  A  should  be  included  in  the  modeled  domain. 
The  current  coupling  approach,  however,  simplifies  this  situation  by  taking  f2  (i.e.,  the  subsur¬ 
face  contribution)  at  the  ADCIRC  boundary  nodes  on  the  coupling  interface. 


Figure  4.  A  side  view  of  the  coupled  pWASH123D-ADCIRC  model. 


Element  searching  is  essential  for  exchanging  data  when  pWASH123D  and  ADCIRC  are 
allowed  to  have  different  mesh  resolutions  on  the  coupling  interface.  Figure  5  explains  the  algo¬ 
rithm  used  in  the  coupler  for  data  exchange.  The  left  plot  depicts  how  the  computed  water  sur¬ 
face  elevation  from  ADCIRC  is  passed  to  pWASH123D  as  the  head- type  boundary  condition  on 
the  coupling  interface.  In  the  left  plot,  the  weights  of  interpolation,  e.g.,  Nn,  Ni„  are  com¬ 
puted  after  element  searching,  where  pWASH123D  node  i  is  found  in  an  ADCIRC  element  com¬ 
prising  nodes  1,  2,  and  3.  With  the  computed  weights  of  interpolation  for  all  of  the  pWASH123D 
nodes  on  the  coupling  interface,  the  right  plot  shows  how  the  pWASH123D  boundary  nodal  flow 
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is  distributed  to  the  associated  ADCIRC  boundary  node  to  form  flux-type  boundary  conditions 
on  the  coupling  interface. 


Figure  5.  Data  exchange  algorithms  used  in  the  coupler  of  the  coupled  model. 


Concurrent  Mode  with  Time  Lagging.  As  shown  in  Figure  6,  pWASH123D  and  ADCIRC 
are  currently  coupled  in  a  concurrent  mode  with  time  lagging.  At  each  coupling  time-step, 
ADCIRC  and  pWASH123D  uses  the  computed  average  boundary  flux  (Qavg)  and  water  surface 
elevation  (H),  both  computed  from  the  previous  coupling  time-step,  as  the  boundary  condition  on 
the  coupling  interface.  Because  of  the  use  of  the  computed  values  from  the  previous  time-step  for 
setting  up  the  boundary  conditions  on  the  coupling  interface,  a  time  lagging  effect  exists  in  the 
current  coupling  mode.  The  coupled  simulation  will  not  proceed  to  the  next  coupling  time-step 
until  both  pWASH123D  and  ADCIRC  have  completed  their  computations  within  one  coupling 
time-step. 

It  is  noted  that  the  current  coupling  of  pWASH123D  and  ADCIRC  requires  the  coupling  time- 
step  to  be  a  common  multiple  of  the  ADCIRC  time-step  and  the  pWASH123D  subsurface  time- 
step.  In  the  pWASH123D  simulation,  it  is  required  that  the  subsurface  time-step  is  a  multiple  of 
the  overland  time-step  and  that  the  overland  time-step  is  a  multiple  of  the  channel  time-step.  As  a 
result,  the  coupling  time-step  is  a  common  multiple  of  the  time-steps  for  ADCIRC,  1-D  channel, 
2-D  overland,  and  3-D  subsurface  computation. 

DEMONSTRATION  EXAMPLE:  A  simplified  Biloxi  model  was  developed  and  used  to  dem¬ 
onstrate  the  coupled  pWASH123D- ADCIRC  model.  Detailed  ADCIRC  modeling  has  been  done 
for  Hurricanes  Katrina  and  Rita  and  extensively  validated  using  a  large  set  of  measured  data 
(Bunya  et  al.  2010a,  2010b).  While  not  suitable  for  highly  detailed  and  accurate  storm  surge 
simulations,  the  ADCIRC  mesh  used  in  this  study  is  appropriate  and  adequate  for  the  purposes  of 
this  work,  which  is  focused  on  test  and  evaluation  of  different  approaches  to  couple  ADCIRC 
and  the  surface-subsurface  model  pWASH123D.  The  domain  of  this  simplified  Biloxi  model 
includes  an  area  covering  the  downstream  Biloxi  River  watershed  in  Mississippi  (the 
pWASH123D  domain,  ~6.9xl08  m2)  and  an  extended  Biloxi  Bay  area  (the  ADCIRC  domain, 
~3.0xl09  m2),  as  sketched  in  Figure  7.  The  blue  lines  in  the  figure  represent  both  rivers/streams 
and  coastal  lines,  while  the  right  plot  shows  the  topographic/bathymetric  variation  within  the 
domain.  The  coastal  line  serves  as  the  coupling  interface  between  ADCIRC  and  pWASH123D. 
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Figure  6.  The  concurrent  mode  with  time  lagging  used  for  the  coupled  pWASH123D-ADCIRC 
model. 


Figure  7.  Biloxi  model  domain:  the  downstream  Biloxi  River 
watershed  in  Mississippi  and  an  extended  Biloxi 
Bay  area. 
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Model  Construction.  In  general,  the  ADCIRC  domain  must  be  large  enough  to  cover  the 
evolution  of  the  storm/hurricane.  The  ADCIRC  domain  shown  in  Figure  7  is  too  small  to  simu¬ 
late  the  Hurricane  Katrina  event  effectively  if  wind  stress  is  the  only  driving  force  applied.  To 
overcome  this  difficulty  without  including  a  large-scale  ADCIRC  domain  in  the  coupled  model, 
we  used  the  computed  water  surface  elevation  from  a  large-scale  ADCIRC  model  to  construct 
the  boundary  condition  on  the  open  boundary.  This  large-scale  ADCIRC  model  was  generated 
and  calibrated  using  the  field  data  provided  by  the  research  group  of  Professor  Shahrouz 
Aliabadi  at  Jackson  State  University  (JSU),  MS.  Figure  8  shows  the  JSU  large-scale  ADCIRC 
model  (lower  left  plot)  as  well  as  the  ADCIRC  mesh  of  the  Biloxi  model  (upper  left  plot),  which 
is  composed  of  20,601  nodes  and  40,470  elements.  The  JSU  large-scale  ADCIRC  model  simu¬ 
lated  the  Hurricane  Katrina  event  over  a  7-day  time  period  until  it  made  landfall  on  the  Gulf 
Coast,  where  the  wind  stress  data  from  NOAA  were  employed.  Our  demonstration  here  consid¬ 
ered  a  72-hour  simulation  starting  from  1800  GMT  August  27  to  1800  GMT  August  30,  2005, 
where  the  storm  surge  peaked  at  around  1400  GMT  August  29,  2005  at  the  west  end  of  the  cou¬ 
pling  interface  (i.e.,  simulation  time  =  44  hr),  which  is  marked  with  a  red  “A”  in  the  upper  left 
plot  of  Figure  8. 

Due  to  time  and  resource  constraints,  the  pWASH123D  model  constructed  for  this  Biloxi  model 
employed  a  simplified  channel  network  (Figure  9),  imposed  homogeneity  to  both  the  overland 
and  subsurface  systems,  and  used  synthetic  rainfall  data  to  set  up  the  model  runs.  Although  the 
model  was  not  calibrated,  it  can  be  used  to  examine  the  coupling  performance  in  BEI-04. 
Figure  10  shows  the  pWASH123D  mesh  that  is  composed  of  301  nodes  and  291  elements  for 
1-D  channel  flow,  15,005  nodes  and  18,982  elements  for  2-D  overland  flow,  and  105,035  nodes 
and  173,892  elements  for  3-D  subsurface  flow  computation.  A  sensitivity  analysis  on  spatial  and 
temporal  resolution  was  conducted  to  determine  the  final  pWASH123D  mesh  and  time-step  sizes 
for  this  study. 

The  pWASH123D  model  boundary  on  the  land  side  was  selected  based  on  the  topographical 
contour,  such  that  drainage  divide  can  be  assumed  and  no-flow  boundary  conditions  can  be 
applied  there.  Figure  1 1  gives  an  overview  of  the  boundary  conditions  employed  in  the  coupled 
model.  The  time-steps  used  for  the  base-case  run  were:  1  sec  for  the  2-D  ADCIRC  computation, 
0.5  sec  for  1-D  channel  flow,  5  sec  for  2-D  overland  flow,  6  min  for  3-D  subsurface  flow,  and 
6  min  for  coupling. 
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ADCIRC  Domain 


Water  Surface  Elevation  Boundary  Condition 
Based  on  A  Large-Scale  Regional  Model 


Generate 

High- 

resolution 

Mesh 


Large-Scale  ADCIRC  Model 


Biloxi  Bay 


Figure  8.  From  the  large-scale  ADCIRC  model  (left  plot)  to  the  ADCIRC  domain  (labeled  in  the  upper 
right  plot)  of  the  Biloxi  model. 
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Figure  9.  Simplified  channel  network  in  the  Biloxi  model. 


Figure  10.  The  pWASH123D  mesh  in  the  Biloxi  model. 
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Water-divide  (no-flow) 
boundary  condition 


Coupled  interface 
boundary  conditions  along 
the  coastal  line 


ADCIRC 


<  Note  >  In  the  “stand-alone”  run, 
the  solution  from  the  JSU  large- 
scale  ADCIRC  model  was  used  to 
set  up  pWASH123D  downstream 
boundary  condition. 


Specified  water  surface 
elevations  from  the  JSU 
large-scale  ADCIRC  model 


Figure  1 1 .  Boundary  conditions  for  the  coupled  model. 


Table  1  lists  the  38  model  runs  conducted  in  this  research  demonstration  to  examine  various 
aspects  of  the  coupled  model,  including  the  scalability  performance,  the  accuracy  performance, 
the  coupling  scheme  effect,  the  time  lagging  effect,  and  the  hydraulic  conductivity  effect.  All  of 
the  model  runs  were  conducted  on  a  Cray  XT3  (Carbon,  ERDC).  These  38  model  runs  include 
stand-alone  (light  green  shading),  one-way  coupling  (light  blue  shading),  and  two-way  coupling 
(yellow  shading)  simulations.  They  are  defined  as  follows: 

(1)  Stand-alone  simulation:  either  the  ADCIRC  or  the  pWASH123D  run  was  conducted. 

(2)  One-way  coupling  simulation:  the  computed  water  surface  elevation  from  ADCIRC  was 
passed  to  pWASH123D  as  the  boundary  condition  on  the  coupling  interface. 

(3)  Two-way  coupling  simulation:  in  addition  to  the  data  transfer  from  ADCIRC  to 
pWASH123D,  the  computed  boundary  flux  along  the  coupling  interface  from 
pWASH123D  was  passed  to  ADCIRC. 

A  no-flow  boundary  condition  was  applied  to  the  coastal  line  (i.e.,  the  coupling  interface)  in  the 
ADCIRC  stand-alone  simulation.  The  computed  water  surface  elevation  from  the  JSU  large- 
scale  ADCIRC  model  was  used  to  set  up  the  boundary  condition  along  the  coastal  line  in  the 
pWASH123D  stand-alone  simulation. 
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Table  1.  Wall  clock  time  of  the  34  model  runs  conducted  in  the  research 
demonstration. 

Run  ID 

No  of  Proc. 

Wall  Clock  Time  (s) 

PE  (%)+ 

Simulation  Type 

pWASH123D 

ADCIRC 

1 

0 

2 

10,230 

Baseline 

ADCIRC  stand  alone 

2 

0 

4 

4,335 

117.99% 

3 

0 

8 

2,624 

97.47% 

4 

0 

16 

1,867 

68.49% 

5 

8 

0 

15,378 

Baseline 

pWASH123D  stand  alone 

6 

16 

0 

9,316 

82.54% 

7 

32 

0 

7,411 

51.88% 

8 

64 

0 

7,559 

25.43% 

9 

8 

0 

19,672 

Baseline 

pWASH123D  stand  alone 

Atsubsurface  —  3  min 

10 

16 

0 

12,369 

79.52% 

11 

32 

0 

10,205 

48.19% 

12 

64 

0 

10,996 

22.36% 

13 

32 

2 

18,094 

Baseline 

Two-Way  Coupling 

14 

32 

4 

9,092 

99.51% 

15 

32 

8 

5,408 

83.64% 

16 

32 

16 

5,067 

44.64% 

17 

32 

2 

18,285 

Baseline 

Two-Way  Coupling,  high  K  (xIO) 

Atsubsurface  —  3  min, 

AtCOupiing  3  min 

18 

32 

4 

9,095 

100.52% 

19 

32 

8 

7,439 

61.45% 

20 

32 

16 

7,650 

29.88% 

21 

32 

2 

19,704 

Baseline 

Two-Way  Coupling,  low  K  (x0.1) 

Atsubsurface  —  3  min, 

Atcoupiing  3  min 

22 

32 

4 

10,516 

93.69% 

23 

32 

8 

7,086 

69.52% 

24 

32 

16 

6,613 

37.24% 

25 

32 

8 

6,107 

N/A 

Two-Way  Coupling,  AtSUbsurface  =  3  min 

26 

32 

2 

18,284 

Baseline 

Two-Way  Coupling, 

Atsubsurface  —  3  min, 

AtCoupiing  3  min 

27 

32 

4 

9,100 

100.46% 

28 

32 

8 

5,921 

77.20% 

29 

32 

16 

6,078 

37.60% 

30 

32 

8 

6,667 

N/A 

One-Way  Coupling,  AtSUbsurface  =  3  min 

31 

32 

2 

11,143 

Baseline 

One-Way  Coupling, 

Atsubsurface  —  3  min, 

AtCOupiing  3  min 

32 

32 

4 

6,931 

80.39% 

33 

32 

8 

6,923 

40.24% 

34 

32 

16 

6,844 

20.35% 

35 

8 

8 

11,430 

Baseline 

Two-Way  Coupling 

36 

16 

8 

6,812 

83.90% 

37 

32 

8 

5,408 

52.84% 

38 

64 

8 

4,943 

28.90% 

*  Time-steps  for  base  case  simulation:  AtADciRc  =  1  s;  Atchannei  =  0.5  s;  AtOVeriand  =  5  s;  AtSUbsurface  =  6  min;  AtCOupiing  =  6  min 
**  Hydraulic  conductivities  for  base  case  simulation:  10  m/hr  (horizontal)  and  1  m/hr  (vertical)  along  channels,  and  100  m/hr 
(horizontal)  10  m/hr  (vertical)  elsewhere 

+  PE  =  parallel  efficiency  =  (NPbasellne  ■Tbasellne)/(NPcomparlson  •r„mpalfso„)-100% 
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RESULTS. 

Scalability  performance.  Table  1  lists  the  wall  clock  times  and  the  computed  parallel  effi¬ 
ciency  (PE)  for  the  38  model  runs,  where  PE  is  defined  in  the  footnote  of  Table  1.  Each  of  the  38 
runs  was  conducted  twice,  and  the  average  wall  clock  times  were  put  in  Table  1.  The  number  of 
the  pWASH123D  processor  is  fixed  at  32  for  all  of  the  one-way  and  two-way  coupling  runs, 
except  for  the  last  four  runs  (i.e.,  Runs  35  -  38).  From  Table  1,  we  observe  (1)  the  PE  value  of 
the  ADCIRC  stand-alone  simulation  is  greater  than  95  percent  up  to  eight  processors;  (2)  the  PE 
value  of  the  pWASH123D  stand-alone  simulation  is  greater  than  77  percent  up  to  16  processors; 
(3)  the  PE  value  of  the  one-way  coupling  simulation  is  greater  than  79  percent  up  to  four 
ADCIRC  processors;  (4)  the  PE  value  of  the  two-way  coupling  simulation  is  greater  than  93 
percent  up  to  four  ADCIRC  processors  and  63  percent  up  to  eight  ADCIRC  processors;  (5)  using 
32  pWASH123D  processors  and  eight  ADCIRC  processors  seems  to  be  the  best  combination  for 
the  designated  72-hour  simulation  when  both  the  PE  and  wall  clock  time  lengths  are  considered. 

Accuracy  performance.  Tables  2  through  4  compare  model  run  results  when  different  num¬ 
bers  of  processors  are  used.  The  model  run  results  include  hourly  solutions  from  zero  to  30  hr, 
every  0.1  hour  from  30  to  60  hr,  and  hourly  again  from  60  to  72  hr.  In  total,  there  are  342  time- 
step  solutions  included.  Table  2  lists  the  comparison  of  1-D  channel  stages,  while  Tables  3  and  4 
are  comparisons  of  the  2-D  overland  water  depth  and  the  ADCIRC  water  surface  elevation, 
respectively.  Four  types  of  deviation  measures  were  employed  for  comparison:  Average  Mean 
Absolute  Deviation  (AMAD),  Average  Root  Mean  Square  Deviation  (ARMSD),  Average  Coef¬ 
ficient  Of  Efficiency  (ACOE),  and  Average  Coefficient  Of  Determination  (ACOD).  They  are 
defined  as  follows: 


f  n  \ 

Ik,-*,, 

i—\ 


it= 1 


AMAD  = 


t 

I 

i<= 1 


ARMSD  = 


Ilk,,- a,,.,  | 

/=! 
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1— <2 - 
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ACOD  = 


it= 1 


~l2 


i= 1 


Z(^-C,)2JZ(K„-^)2 


i= 1 


i'=l 


(4) 


where  «  =  the  number  of  comparisons  between  the  values  from  the  reference  and  the  compara¬ 
tive  groups,  both  are  computed  here,  within  each  time-step  ;  nt  =  the  number  of  time-steps 
included  for  data  comparison;  cit  i  =  the  i-th  value  for  comparison  from  the  comparative  group 
that  is  associated  with  the  it-th  time-step;  =  the  i-th  value  for  comparison  from  the  reference 

group  that  is  associated  with  the  i-th  time-step;  Cu  =  the  mean  value  over  n  comparisons  in  the 

comparative  group  at  the  it-th  time-step;  and  Ru  =  the  mean  value  over  n  comparisons  in  the 
reference  group  at  the  it-th  time-step.  Basically,  the  two  groups  of  data  are  in  close  agreement 
when  AMAD  and  ARMSD  are  small  and  ACOE  and  ACOD  are  close  to  unity,  or  (i  -  ACOE)  and 
(7  -ACOD)  are  close  to  zero. 

It  is  observed  from  Tables  2  through  4  that  both  (1  -  ACOE)  and  ( 1  -ACOD)  are  close  to  zero  in 
all  comparisons,  and  that  AMAD  and  ARMSD  are  small.  The  values  of  the  deviation  measures 
shown  in  these  three  tables  indicate  that  the  coupled  pWASH123D-ADCIRC  model  provides 
consistent  computational  results  in  stand-alone,  one-way  coupling,  and  two-way  coupling  simu¬ 
lations  when  the  number  of  processors  varies. 


Table  2.  Comparison  statistics  of  the  computed  1-D  channel  stage  (n  =  301  and 
nt  =  342) 

Comparison  ID 

Compared  Model  Runs 

1-ACOD 

1-ACOE 

ARMSD ,  m 

AMAD ,  m 

Reference  Run 

Comparative  Run 

SA1 

Run  11 

Run  9 

2.60E-05 

5.90E-05 

1.79E-04 

1.00E-04 

SA2 

Run  11 

Run  10 

2.80E-05 

6.20E-05 

1 .83E-04 

1.01E-04 

SA3 

Run  11 

Run  12 

3.80E-05 

8.70E-05 

2.09E-04 

1.18E-04 

SA4 

Run  7 

Run  5 

1.20E-05 

2.50E-05 

1 .23E-04 

6.15E-05 

SA5 

Run  7 

Run  6 

4.20E-05 

9.40E-05 

2.43E-04 

1.37E-04 

SA6 

Run  7 

Run  8 

5.70E-05 

1 .44E-04 

3.38E-04 

1 .96E-04 

IWayl 

Run  33 

Run  31 

3.10E-05 

6.30E-05 

2.27E-04 

1.08E-04 

1Way2 

Run  33 

Run  32 

3.00E-05 

6.50E-05 

2.28E-04 

1.2  IE-04 

1Way3 

Run  33 

Run  34 

3.90E-05 

8.20E-05 

2.76E-04 

1.40E-04 

2Way1 

Run  28 

Run  26 

1.15E-04 

3.07E-04 

1.99E-03 

1.43E-03 

2Way2 

Run  28 

Run  27 

7.20E-04 

2.09E-03 

4.99E-03 

3.56E-03 

2Way3 

Run  28 

Run  29 

3.89E-04 

1.1  IE-03 

4.1  IE-03 

3.01  E-03 

2Way4 

Run  15 

Run  13 

1.01E-04 

3.28E-04 

1.29E-03 

9.73E-04 

2Way5 

Run  15 

Run  14 

5.40E-04 

1.70E-03 

3.22E-03 

2.47E-03 

2Way6 

Run  15 

Run  16 

1.63E-04 

5.24E-04 

1.73E-03 

1.29E-03 
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Table  3.  Comparison  statistics  of  the  computed  2-D  overland  water  depth  (n  = 
15,005  and  nt  =  342) 

Comparison  ID 

Compared  Model  Runs 

1-ACOD 

1-ACOE 

ARMSD 

AMAD 

Reference  Run 

Comparative  Run 

Run  11 

Run  9 

5.70E-09 

1.16E-08 

4.52E-05 

8.42E-06 

Run  11 

Run  10 

4.86E-09 

9.86E-09 

4.26E-05 

7.91  E-06 

Run  11 

Run  12 

2.03E-06 

4.09E-06 

7.63E-05 

1.15E-05 

SA4 

Run  7 

Run  5 

2.41  E-09 

4.87E-09 

3.14E-05 

5.20E-06 

SA5 

Run  7 

Run  6 

9.05E-09 

1.85E-08 

6.07E-05 

1.12E-05 

SA6 

Run  7 

Run  8 

2.40E-08 

4.98E-08 

9.27E-05 

1.85E-05 

IWayl 

Run  33 

Run  31 

1.03E-08 

2.07E-08 

5.32E-05 

7.03E-06 

Run  33 

Run  32 

1.09E-08 

2.20E-08 

5.98E-05 

9.73E-06 

Run  33 

Run  34 

1.2  IE-08 

2.44E-08 

6.34E-05 

9.55E-06 

2Way1 

Run  28 

Run  26 

2.65E-06 

5.42E-06 

8.49E-04 

2.82E-04 

2Way2 

Run  28 

Run  27 

1 .47E-05 

3.09E-05 

2.21  E-03 

8.16E-04 

2Way3 

Run  28 

Run  29 

8.80E-06 

1.86E-05 

1.68E-03 

6.20E-04 

2Way4 

Run  15 

Run  13 

1.12E-06 

2.32E-06 

4.76E-04 

1.52E-04 

2Way5 

Run  15 

Run  14 

5.03E-06 

1.05E-05 

1.16E-03 

3.78E-04 

2Way6 

Run  15 

Run  16 

1.89E-06 

3.90E-06 

6.66E-04 

2.09E-04 

Table  4.  Comparison  statistics  of  the  computed  2-D  ADCIRC  water  surface 
elevation  (n  =  20,601  and  nt  =  342) 

Comparison  ID 

Compared  Model  Runs 

1-ACOD 

1-ACOE 

ARMSD 

AMAD 

Reference  Run 

Comparative  Run 

mmmaam 

Run  3 

Run  1 

0.00E+00 

0.00E+00 

3.13E-14 

2.19E-16 

Run  3 

Run  2 

0.00E+00 

0.00E+00 

2.26E-13 

1.58E-15 

SA3 

Run  3 

Run  4 

0.00E+00 

0.00E+00 

1.82E-13 

1.40E-15 

IWayl 

Run  33 

Run  31 

0.00E+00 

0.00E+00 

3.13E-14 

2.19E-16 

1Way2 

Run  33 

Run  32 

0.00E+00 

0.00E+00 

2.26E-13 

1.58E-15 

1Way3 

Run  33 

Run  34 

0.00E+00 

0.00E+00 

1.82E-13 

1.40E-15 

2Way1 

Run  28 

Run  26 

4.54E-07 

9.1  IE-07 

2.67E-04 

9.69E-05 

2Way2 

Run  28 

Run  27 

3.99E-06 

7.99E-06 

7.84E-04 

3.03E-04 

2Way3 

Run  28 

Run  29 

2.19E-06 

4.38E-06 

5.75E-04 

2.32E-04 

2Way4 

Run  15 

Run  13 

8.30E-08 

1.67E-07 

1.18E-04 

4.15E-05 

2Way5 

Run  15 

Run  14 

5.58E-07 

1.12E-06 

2.83E-04 

9.72E-05 

2Way6 

Run  15 

Run  16 

2.04E-07 

4.09E-07 

1 .86E-04 

6.34E-05 

Comparison  among  coupling  schemes.  We  compare  results  from  the  model  runs  associ¬ 
ated  with  stand-alone,  one-way  coupling,  and  two-way  coupling  simulations  when  AtADCiRC  = 
1  s,  Atchannei  —  0-5  s,  At0veriand  —  5  s,  Atsubsurface  —  3  min,  AtCOupiing  —  3  min,  and  the  number  of 
processors  are  8  and  32  for  ADCIRC  and  pWASH123D,  respectively.  That  is,  Run  3  (ADCIRC 
stand-alone),  Run  1 1  (pWASH123D  stand-alone),  Run  33  (one-way  coupling),  and  Run  28  (two- 
way  coupling)  are  compared. 
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First,  we  compare  the  wall  clock  time  of  Runs  3,  11,  33,  and  28  that  can  be  extracted  from 
Table  1  (Table  5).  It  is  clear  that  when  the  number  of  processors  was  set  to  32  for  pWASH123D 
and  8  for  ADCIRC,  the  performance  bottleneck  was  on  the  pWASH123D  computation  for  this 
Biloxi  example.  It  should  be  noted  that  the  wall  clock  time  for  two-way  coupling  (Run  28)  is  less 
than  that  for  one-way  coupling  (Run  33)  and  much  less  than  that  for  pWASH123D  stand-  alone 
(Run  11),  though  the  pWASH123D  processors  were  the  same  (i.e.,  32)  for  these  three  runs.  As 
we  should  expect,  different  coupling  schemes  will  result  in  different  water  surface  elevation  val¬ 
ues  on  the  coupling  interface.  While  pWASH123D  solves  nonlinear  1-D  channel,  2-D  overland, 
and  3-D  subsurface  equations,  different  water  surface  elevation  values  applied  to  the  coastal  line 
(i.e.,  the  coupling  interface)  may  require  different  numbers  of  nonlinear  iterations  to  reach  con¬ 
vergent  solutions.  The  result  shown  in  Table  5  suggests  that  the  coupling  simulation  can  be  com¬ 
putationally  more  efficient  than  the  pWASH123D  stand-alone  simulation  with  only  a  few  extra 
processors  employed  for  ADCIRC  computation. 


Table  5. 

Wall  clock  time  of  Runs  3,  11,  33,  and  28. 

Run  ID 

No  of  Proc. 

Wall  Clock  Time  (s) 

Simulation  Type 

pWASH123D 

ADCIRC 

3 

0 

8 

2,624 

ADCIRC  stand  alone 

11 

32 

0 

10,205 

pWASH123D  stand  alone 

33 

32 

8 

6,923 

One-Way  Coupling 

28 

32 

8 

5,921 

Two-Way  Coupling 

*  Time-steps  for  base  case  simulation:  AtADciRc  =  1  s;  Atchannei  =  0.5  s;  Atoveriand  =  5  s;  AtSUbsurface  =  3  min;  AtC0Up|ing  =  3  min 
**  Hydraulic  conductivities  for  base  case  simulation:  100  m/hr  (horizontal)  and  10  m/hr  (vertical) 

Tables  6  through  8  compare  model  run  results  among  Runs  3,  1 1,  33,  and  28.  The  values  of  the 
four  deviation  measures  are  much  greater  than  those  shown  in  Tables  2  through  4,  indicating 
discrepancies  between  the  compared  model  runs  due  to  different  coupling  schemes.  The  four 
deviation  measures  have  the  smallest  values  associated  with  the  computed  2-D  ADCIRC  water 
surface  elevation  and  the  largest  values  with  the  computed  1-D  channel  stage,  indicating  that 
when  compared  with  the  2-D  overland  water  depth  and  the  2-D  ADCIRC  water  surface  eleva¬ 
tion,  the  1-D  channel  stage  is  more  sensitive  to  the  switch  from  a  coupling  scenario  (i.e.,  stand- 
along,  one-way  coupling,  or  two-way  coupling)  to  another. 


Table  6.  Comparison  statistics  of  the  computed  1-D  channel  stage  among  stand¬ 
alone,  one-way  coupling,  and  two-way  coupling. 

Comparison  ID 

Compared  Model  Runs 

1-ACOD 

1-ACOE 

ARMSD,  m 

AMAD,  m 

Reference  Run 

Comparative  Run 

2Way-SA 

Run  28 

Run  11 

6.21  E-02 

7.66E-01 

1.03E-01 

9.35E-02 

2Way-1Way 

Run  28 

Run  33 

5.89E-02 

7.00E-01 

9.31  E-02 

8.44E-02 

IWay-SA 

Run  33 

Run  11 

9.54E-03 

1.67E-01 

3.00E-02 

2.63E-02 
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Table  7.  Comparison  statistics  of  the  compi 
stand-alone,  one-way  coupling,  and  two-wa; 

ited  2-D  overland  water  depth  among 
/  coupling. 

Comparison  ID 

Compared  Model  Runs 

1-ACOD 

1-ACOE 

ARMSD,  m 

AMAD,  m 

Reference  Run 

Comparative  Run 

2Way-SA 

Run  28 

Run  11 

2.22E-03 

6.38E-03 

3.82E-02 

1.66E-02 

2Way-1Way 

Run  28 

Run  33 

1.66E-03 

4.32E-03 

3.34E-02 

1.44E-02 

IWay-SA 

Run  33 

Run  11 

6.47E-04 

1.92E-03 

1.37E-02 

5.78E-03 

Table  8.  Comparison  statistics  of  the  computed  2-D  ADCIRC  ' 
elevation  among  stand-alone,  one-way  coupling,  and  two-wa] 

water  surface 
/  coupling. 

Comparison  ID 

Compared  Model  Runs 

1-ACOD 

1-ACOE 

ARMSD i,  m 

AMAD ,  m 

Reference  Run 

Comparative  Run 

2Way-SA 

Run  28 

Run  3 

3.16E-04 

7.13E-04 

8.48E-03 

3.17E-03 

2Way-1Way 

Run  28 

Run  33 

3.16E-04 

7.13E-04 

8.48E-03 

3.17E-03 

IWay-SA 

Run  33 

Run  3 

0.00E+00 

0.00E+00 

0.00E+00 

0.00E+00 

Figures  12  and  13  depict  the  absolute  differences  of  1-D  channel  stage  between  model  runs  11 
(pWASH123D  stand  alone),  33  (one-way  coupling),  and  28  (two-way  coupling).  Figure  14  plots 
the  absolute  differences  of  2-D  overland  water  depth  between  the  aforementioned  model  runs. 
Figure  12  shows  that  significant  water  stage  differences  exist  between  two-way  coupling  and  the 
other  two  simulations  during  the  storm  period  (36  -  56  hr).  Channel  water  stage  differences 
become  minimal  during  quiet  periods  (0-36  and  56  -  72  hr).  Moreover,  upstream  reaches  have 
smaller  stage  differences  than  downstream  reaches  based  on  the  channel  reach  distribution  speci¬ 
fied  in  Figure  9.  Figure  13  shows  that  stage  differences  at  Junctions  1  and  3  (Figure  9)  between 
two-way  coupling  and  stand-alone  simulations  are  insignificant  during  quiet  periods  but  can 
reach  0.6  m  during  the  storm  period.  Figure  14  shows  that  the  differences  of  overland  water 
depth  increased  as  the  storm  came  and  decreased  as  the  storm  left.  All  the  three  figures  suggest 
that  the  coupling  scheme  has  limited  effect  on  watershed  simulation  results  during  non- storm 
periods.  The  discrepancies  between  the  channel  stage  results  from  two-way  and  one-way  cou¬ 
pling  (Figure  12b)  increase  with  the  strength  of  the  storm,  suggesting  that  the  greatest  watershed- 
nearshore  interaction  is  at  the  storm  peak,  which  implies  that  one-way  coupling  will  be  insuffi¬ 
cient  in  accounting  for  watershed-nearshore  interaction  when  storms  are  greater  than  a  certain 
degree. 
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Figure  12.  Absolute  differences  of  1-D  channel  stage  between  (a)  two-way  coupling  and  stand-alone, 
(b)  two-way  coupling  and  one-way  coupling,  and  (c)  one-way  coupling  and  stand-alone 
simulations  at  various  times,  where  Reach  R-1  contains  Nodes  1-36,  R-2  Nodes  37-88, 
R-3  Nodes  89-108,  R-4  Nodes  109-112,  R-5  Nodes  1 13-173,  R-6  Nodes  174-196, 

R-7  Nodes  197-210,  R-8  Nodes  211-261,  R-9  Nodes  262-286,  R-10  Nodes  287-301. 


Figure  13.  Absolute  differences  of  1-D  channel  stage  (in  meter)  between  two-way  coupling  and  stand¬ 
alone  simulations  at  Junctions  1  (left)  and  3  (right). 
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(e) 

Figure  14.  Absolute  differences  of  overland  water  depth  between  two-way  coupling  and  stand-alone 
simulation  at  the  (a)  30th,  (b)  40th,  (c)  45th,  (d)  50th,  and  (e)  60th  hr. 


Figure  15  presents  the  absolute  differences  of  2-D  ADCIRC  water  surface  elevation  between 
two-way  coupling  (Run  28)  and  stand-alone  (Run  3)  simulations  at  various  times.  Significant 
differences  occur  in  the  Biloxi  Bay  during  the  storm  event  (Time  =  45  hr).  These  differences 
decrease  with  the  distance  from  the  coastal  line,  indicating  that  the  impact  from  pWASH123D  to 
ADCIRC  was  mainly  in  the  bay  area  and  around  the  coastal  line. 
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(e) 

Figure  15.  Absolute  differences  of  ADCIRC  water  surface  elevation  (in  meter)  between  two-way  coupling 
and  stand-alone  simulations  at  the  (a)  30th,  (b)  40th,  (c)  45th,  (d)  50th,  and  (e)  60th  hr. 


Effect  of  time  lagging.  Figures  16  and  17  present  the  absolute  differences  of  overland  water 
depth  and  ADCIRC  water  surface  elevation  between  3-min  (Run  28)  and  6-min  (Run  25)  cou¬ 
pling.  The  time  lagging  effect  was  insignificant  except  for  at  around  the  peak  of  the  storm  (i.e., 
time  =  45  hr).  The  large  lagging  effect  occurred  mainly  around  channel  exits,  especially  the  exit 
of  Biloxi  River. 
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(c) 

Figure  16.  Absolute  differences  of  overland  water  depth  (in  meter)  between  3-minute  coupling  and 
6-minute  coupling  at  the  (a)  40th,  (b)  45th,  and  (c)  50th  hr. 


(c) 

Figure  17.  Absolute  differences  of  ADCIRC  water  surface  elevation  (in  meter)  between  3-minute  coupling 
and  6-minute  coupling  at  the  (a)  40th,  (b)  45th,  and  (c)  50th  hr. 
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Effect  of  hydraulic  conductivity.  For  a  demonstration  purpose,  there  were  only  two  subsur¬ 
face  materials  considered  in  this  simplified  Biloxi  model:  fine  sand  along  channels  and  sand 
elsewhere.  The  horizontal/vertical  hydraulic  conductivities  (K)  for  sand  and  fine  sand  were  set  to 
100/10  and  10/1  m/hr,  respectively,  for  the  base  case  (Table  1)  and  the  subsurface  domain. 
Figure  18  shows  the  differences  of  ADCIRC  water  surface  elevation  between  the  base  case  (i.e., 
Run  28)  and  when  the  K  values  are  10  times  those  of  the  base  case  (i.e.,  Run  19).  On  the  other 
hand,  Figure  19  presents  the  water  surface  elevation  differences  between  the  base  case  and  when 
the  K  values  are  1/10  of  the  base  case  (i.e.,  Run  23).  When  K  is  10  times  those  of  the  base  case 
values,  large  differences  of  water  surface  elevation  exist  in  the  bay  area  during  the  storm  period. 
This  is  because  larger  K  directs  to  greater  surface-subsurface  interaction,  which  promotes  sig¬ 
nificant  water  exchange  between  watershed  and  nearshore  models  when  a  storm  is  the  driving 
force  for  water  movement.  On  the  other  hand,  smaller  K  implies  less  surface-subsurface  interac¬ 
tion  and  a  smaller  contribution  to  the  flux  across  the  coupling  interface  of  the  watershed- 
nearshore  model. 


(e) 

Figure  18.  Absolute  differences  of  ADCIRC  water  surface  elevation  (in  meter)  between  Run  28  (base 


case,  K  =  Kbase)  and  Run  19  (K  =  10xKbase)  at  the  (a)  40th,  (b)  43rd,  (c)  45th,  (d)  47th,  and 
(e)  50th  hr. 


22 


ERDC  TN-SWWRP-1 0-2 
April  2010 


(e) 

Figure  19.  Absolute  differences  of  ADCIRC  water  surface  elevation  (in  meter)  between  Run  28  (base 

case,  K  =  Kbase)  and  Run  23  (K  =  0.1xKbase)  at  various  times:  40,  43,  45,  47,  and  50  hr  (left  to 
right). 


SUMMARY  AND  FUTURE  PLAN:  In  this  technical  note,  a  coupled  watershed-nearshore 
model  newly  developed  in  the  BEI  was  presented  as  a  research  demonstration  project  under  the 
support  of  SWWRP.  The  model  coupled  pWASH123D  and  ADCIRC  through  ESMF  and 
DBuilder.  The  pWASH123D  and  ADCIRC  models  were  currently  coupled  in  a  concurrent 
model  with  time  lagging,  and  the  coupling  interface  was  the  coastal  line.  A  simplified  Biloxi  Bay 
model  was  constructed  for  demonstration.  This  Biloxi  Bay  model  used  historical  data  as  well  as 
the  simulation  result  from  a  large-scale  2-D  ADCIRC  simulation  of  hurricane  Katrina  as  the 
driving  forces  for  water  movement.  GMS  6.5  was  used  to  produce  the  pWASH123D  input  files 
and  the  ADCIRC  domain  geometry  (mesh)  files.  By  using  GMS,  we  can  ensure  that  the 
pWASH123D  and  ADCIRC  interface  boundary  nodes  truly  sit  on  the  coupling  interface,  which 
is  essential  for  element  searching. 
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This  task  included  38  model  runs  to  examine  parallel  efficiency,  accuracy,  and  impacts  from 
coupling  scheme,  time  lagging  method,  and  assumed  hydraulic  conductivity.  Through  the  analy¬ 
sis  of  the  model  run  results,  the  following  were  observed. 

•  The  most  significant  impact  from  coupling  ADCIRC  to  pWASH123D  was  on  the  areas 
around  the  coastal  line  near  shallow  water  areas  and  the  channels. 

•  The  most  significant  impact  from  coupling  pWASH123D  to  ADCIRC  was  on  the  bay 
area  and  around  the  coastal  line. 

•  The  time  lagging  effect  became  significant  during  storm  events,  suggesting  that  higher 
coupling  frequency  should  be  used  during  these  periods  of  time. 

•  Large  hydraulic  conductivity  implies  great  surface-subsurface  interaction  and  subse¬ 
quently  promotes  water  exchange  between  the  watershed  and  nearshore  models. 

•  When  compared  with  the  2-D  overland  water  depth  and  the  2-D  ADCIRC  water  surface 
elevation,  the  1-D  channel  stage  is  more  sensitive  to  coupling  scenarios  (i.e.,  stand-alone, 
one-way  coupling,  or  two-way  coupling). 

•  Two-way  coupling  is  necessary  to  accurately  account  for  watershed-nearshore  interaction 
during  storm  events. 

•  The  coupled  pWASH123D-ADCIRC  model  provides  consistent  computational  results  in 
stand-alone,  one-way  coupling,  and  two-way  coupling  simulations  when  the  number  of 
processors  varies. 

•  The  coupling  simulation  can  be  computationally  more  efficient  than  the  pWASH123D 
stand-alone  simulation  with  only  a  few  extra  processors  employed  for  ADCIRC  compu¬ 
tation  (e.g.,  two-way  coupling  Runs  15  and  28  versus  stand-alone  Runs  7  and  11,  respec¬ 
tively).  In-depth  investigation  for  this  is  needed  to  determine  the  reason(s). 

•  For  this  simplified  Biloxi  Bay  model,  using  32  pWASH123D  processors  and  8  ADCIRC 
processors  was  the  best  combination  for  the  designated  72-hour  simulation  when  both  the 
PE  and  wall  clock  time  are  considered. 

The  coupled  model  in  the  current  form  can  be  used  to  model  water  flow  in  combined  watershed- 
nearshore  systems.  Using  high  performance  computing  produces  the  spatial  distributions  of 
water  flow  and  head  in  the  combined  system  at  designated  time-steps.  The  coupled  model  can 
help  better  understand  both  the  surface- subsurface  and  the  watershed-nearshore  interactions  as 
well  as  construct  and  verify  adequate  plans  for  water  management  and  other  water-related  envi¬ 
ronmental  issues. 

The  coupled  pWASH123D- ADCIRC  model  is  the  first  coupled  unstructured-mesh  model  using 
ESMF.  Features  and  capabilities  developed  for  this  coupled  model  can  also  be  applied  to  the 
coupling  of  structured-  and  unstructured-mesh  models.  When  compared  to  stand-alone  models, 
the  coupled  model  exhibits  more  accurate  boundary  conditions  applied  to  the  interface  boundary 
between  the  watershed  and  the  coastal  models,  especially  when  surface-subsurface  and  water¬ 
shed-coastal  ocean  interactions  are  significant  during  storm  events.  As  a  result,  the  hydro-system 
in  the  nearshore  area  has  been  modeled  better,  and  an  improved  understanding  of  the  complex 
interactions  can  be  obtained.  The  future  improvement/development  may  include  the  following. 
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•  Incorporate  temporal  interpolation  to  reduce  the  time  lagging  effect. 

•  Implement  the  coupling  through  overlapped  wetting-drying  areas  and  extend  the  subsur¬ 
face  domain  to  some  distances  off  the  coastal  line. 

•  Develop  salinity  transport  and  density-dependent  flow  capabilities  in  the  coupled  model. 

•  Make  the  watershed-nearshore  coupling  time- step  a  factor,  rather  than  a  multiple,  of  the 
3-D  subsurface  time-step. 

ADDITIONAL  INFORMATION: 

This  work  was  funded  by  the  SWWRP.  At  the  time  of  publication,  the  SWWRP  Program  Man¬ 
ager  was  Dr.  Steven  L.  Ashby:  Steven.L.Ashby@usace.army.mil.  Dr.  Jeffery  Allen  (ITL)  and 
Aaron  Byrd  reviewed  this  technical  note. 

This  technical  note  should  be  cited  as  follows: 

Cheng,  H.-P.,  J.-R.  C.  Cheng,  R.  M.  Hunter,  and  H.-C.  Lin.  2010.  Demonstration 
of  a  coupled  watershed-nearshore  model.  ERDC  TN-SWWRP-1 0-XX.  Vicks¬ 
burg,  MS:  U.S.  Army  Engineer  Research  and  Development  Center,  https ://swwrp. 
us  ace.  army.  mil/. 
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